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Abstract 

Studies on computer-aided instruction and web-based learning have left many questions unanswered about the 
most effective use of technology-assisted education in graduate medical education. 

Objective: We conducted a review of the current medical literature to report the techniques, methods, frequency 
and effectiveness of technology-assisted education in graduate medical education. 

Methods: A structured review of MEDLINE articles dealing with "Computer-Assisted Instruction," "Internet or World 
Wide Web," "Education" and "Medical" limited to articles published between 2002-2007 in the English language 
was performed. RESULTS: The two literature searches returned 679 articles; 184 met our inclusion and exclusion 
criteria. In 87 articles, effectiveness was measured primarily using self-reported results from a survey of subjects. 
Technology-assisted education was superior to traditional methods in 42 of the 64 direct comparison articles (66%, 
95% CI 53-77%). Traditional teaching methods were superior to technology-assisted education in only 3/64 (5%, 
95% CI 1-13%). The remaining 19 direct comparison articles showed no difference. A detailed review of the 64 
comparative studies (technology-assisted education versus traditional teaching methods) also failed to identify a 
best method or best uses for technology-assisted education. 

Conclusions: Technology-assisted education is used in graduate medical education across a variety of content 
areas and participant types. Knowledge gain was the predominant outcome measured. The majority of studies that 
directly compared knowledge gains in technology-assisted education to traditional teaching methods found 
technology-assisted education equal or superior to traditional teaching methods, though no "best methods" or 
"best use" was found within those studies. Only three articles were specific to Emergency Medicine, suggesting 
further research in our specialty is warranted. 

Keywords: education, medical, graduate, computer-assisted instruction, Internet or World Wide Web, simulation, 
virtual reality 



Background 

For decades, medical educators have looked for ways to 
use computer technology to assist education. In the late 
1960s, pioneer medical educators began to develop com- 
puter systems that laid the foundation for computer- 
assisted instruction in medical education [1,2]. These 
early systems consisted of drill and practice questions 
and later basic true-false or matching questions. As 
computer technology improved, so did computer- 
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assisted instruction. Over time, rudimentary computer- 
aided instruction systems were augmented with multi- 
media laden systems rich with sound, video and 
animation 

The Internet ushered in a new era that allowed for 
easy distribution of material, easy access by students 
and central management by administrators [3,4]. Tech- 
nologies such as simulation and virtual reality were 
developed that added new dimensions to instruction. 
Today, computer-assisted instruction, web-based educa- 
tion simulation and now virtual reality are some of the 
technologies frequently used to support graduate 
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medical education. We refer to these methods as tech- 
nology-assisted education. 

Multiple studies have been performed to evaluate tech- 
nology-assisted education in medical education. In a 
1992 meta-analysis, Cohen et al. found a "medium-sized 
effect" of computer-assisted instruction on student learn- 
ing and recommended more research to identify specific 
features of computer-assisted instruction that lead to 
improved student performance [5]. In a structured review 
published in 2002, Chumley-Jones et al. found that web- 
based learning (WBL) methods can result in student 
gains but cautioned "...It is a valuable addition to our 
educational armory, but it does not replace traditional 
methods. ...Educators must define WBL's unique educa- 
tional contribution. "[6] In a 2006 structured review, 
Cook stated that "Research on WBL in medical education 
has done little to inform practice." [7] 

The questions of when, where and how to best use 
technology- assisted education have not been adequately 
addressed by the existing literature. As new technologies 
emerge, new questions continually arise, further compli- 
cating matters. Given the cost in time and money asso- 
ciated with the use of many technology-assisted 
education systems, lack of knowledge on how to best 
use this technology places educators in a position of 
dual jeopardy. Valuable resources could be wasted, and 
potentially more important, ineffective instructional 
methods could be unintentionally implemented. Emer- 
gency Medicine (EM) educators have to navigate these 
complicated issues when trying to determine the role of 
technology-assisted education in their curriculum. EM 
educators, in particular, are hampered by the relative 
paucity of EM specific studies and must therefore rely 
on the pool of information present in the general medi- 
cal education literature. We examined the current tech- 
nology-assisted education-related medical literature to 
determine the scope of use of technology-assisted edu- 
cation, whether technology-assisted education improved 
knowledge when compared with traditional teaching 
methods, and whether a "Best Method or Best Use" for 
technology-assisted education could be identified. 

Our objectives were to report the techniques, methods 
and frequency of use of technology-assisted education in 
graduate medical education, to evaluate the effectiveness 
of technology-assisted education in improving knowl- 
edge compared to traditional and lecture-based teaching 
methods, and to determine if there was a consensus or 
general agreement on a "Best Method or Best Use" for 
technology-assisted education that could be identified. 

Materials and methods 

Design 

We performed a structured review of the medical litera- 
ture on technology-assisted education. 



Search strategies 

Two searches were completed using the National 
Library of Medicine's PubMed database. The first was 
performed by the lead author and combined the follow- 
ing keywords using the Boolean search term AND: 
"Computer-Assisted Instruction," "Internet or World 
Wide Web," "Education" and "Medical." The search was 
limited to articles published in the last 5 years in the 
English language. The 5-year time period was chosen to 
focus on current teaching method technologies. This 
search was completed on 30 October 2007 and resulted 
in 271 citations. The second search was completed by 
the Information Services librarian using the MeSH 
terms ("Education, Medical" OR "Education, Medical, 
Undergraduate" OR "Education, Medical, Graduate" OR 
"Education, Medical, Continuing") AND "Computer- 
Assisted Instruction." This search was limited to studies 
published in the past 5 years in the English language 
and performed on 6 December 2007, resulting in 408 
citations. 

Article selection 

We included all studies that involved graduate medical 
education and computer-assisted instruction, web-based 
education, simulation, virtual reality or other technolo- 
gies. Evaluative articles were defined as those articles 
that conducted an evaluation of the education effective- 
ness of the technology or process. We excluded descrip- 
tive articles (defined as those that described a 
technology or process but did not assess its educational 
effectiveness), as well as dental, veterinary, podiatry and 
patient education articles. 

Article review process 

Two investigators conducted a primary review of each 
article to determine if they were evaluative or descrip- 
tive. A third author resolved discrepancies. 

Articles underwent a secondary review to determine 
the method of assessment used to determine effective- 
ness and collection of other data elements. Group one 
articles were defined as those that conducted no com- 
parison between educational methods and determined 
effectiveness through survey/subject self-report. Group 
two articles were defined as those that conducted no 
comparison between educational methods and deter- 
mined effectiveness using some type of objective before 
and after measurement. Group three articles were 
defined as those in which a comparison between educa- 
tional methods was done and effectiveness was mea- 
sured using an objective method such as a pretest and 
posttest, checklist, computer log or direct observation. 
Articles were also reviewed for any information that 
suggested a proven or generally accepted best method 
was used in the study. During secondary review, if 
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objective assessment methods were not present, articles 
were considered descriptive and excluded. 

Data collection 

A data collection sheet was developed and pretested on 
three faculty members who provided feedback on clarity 
and general usability. Study investigators were then 
instructed on how to complete the article reviews using 
the data collection sheet. The five study investigators 
who performed the article reviews completed a pilot 
review using the data collection sheet and eight ran- 
domly chosen articles from the study sample. The pilot 
review provided the opportunity to clarify items on the 
data sheet and article review methods. Feedback from 
this pilot review was used to further modify the data 
collection sheet and article review methods. 

All articles meeting inclusion criteria were then 
reviewed by a study investigator and data elements 
were recorded for each article. If the reviewing investi- 
gator had any questions about a data element, the arti- 
cle was reviewed by a panel of investigators consisting 
of the lead author and two additional investigators. 
The coding of the data element in question was 
resolved by the majority opinion of this panel. Data 
were entered by a research technician into a Microsoft 
Access database. 

Data analysis 

Data were analyzed using Stata RJ , version 11. Data are 
presented using descriptive statistics (means and propor- 
tions) with associated 95% confidence intervals (CI). 

Results 

The results of the searches and initial review for eligibil- 
ity are shown in Figure 1. From the 679 studies origin- 
ally identified in the searches, 257 articles were excluded 
because of duplication or failure to meet study criteria 
in the primary review process. During the secondary 
review, 238 articles were excluded because of nonobjec- 
tive assessment methods or not meeting inclusion cri- 
teria. A total of 184 studies met the inclusion criteria 
and were reviewed by an investigator. Descriptive data 
from these studies are shown in Table 1, Of these, 87 
articles were group 1 where no comparison was done 
between educational methods and effectiveness was 
measured primarily using self-reported results from a 
survey of subjects; 18 articles were group 2 where no 
comparison was done between educational methods and 
objective before and after methods where used to assess 
effectiveness. There were 79 articles in group 3 (43%, 
95% CI 36-50), which consisted of studies in which a 
comparison was conducted between educational meth- 
ods and objective methods were used to measure effec- 
tiveness (Table 2). Assessment methods commonly used 



1st Literature Search 
271 Studies 



2nd Literature Search 
408 Studies 



679 Total Studies 



Primary Review 
257 Ouplicate or 
Didrrt Meet Inclusion Criteria 



422 Studies 



Secondary Review 
233 studies E«luoed 



184 Studies Reviewed 



Figure 1 Article selection and review process 



included subject self-assessment by survey, computer 
log, a pretest and posttest, checklists and direct 
observation. 

In 64 of the group three articles (64/184, 35%, 95% CI 
28-42), there was a direct comparison between technol- 
ogy-assisted education and traditional teaching methods 
(Table 3). In the majority of these 64 articles, the sub- 
jects were medical students and content area was clini- 
cal medicine. Technology-assisted education was 
superior to traditional teaching methods in 42 of the 64 
direct comparison articles (66%, 95% CI 53-77%), tradi- 
tional teaching methods were superior to technology- 
assisted education in only 3 of the 64 articles (5%, 95% 
CI 1-13%), and the remaining 19 showed no difference. 
No consistent best methods or best uses were identified 
after review of the articles. A detailed review of the 64 
comparative studies (technology-assisted education ver- 
sus traditional teaching methods also failed to identify a 
best method or best uses for technology-assisted 
education. 

Most articles evaluated technology-assisted education 
with regard to clinical medicine (123/184, 67%, 95% CI 
59-74%) and basic science education (44/184, 24%, 95% 
CI 18-31%). Knowledge gains were the most common 
outcome assessed by the literature (90/184, 49%, 95% CI 
42-56%). Other outcomes commonly assessed included 
satisfaction (82/184, 45%, 95% CI 37-52%), clinical skills 
(57/184, 31%, 95% CI 24-38%), attitudes (47/184, 26% 
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Table 1 Descriptive study data 

Proportions % (95% CI) 



95% CI 19-32%) and access to technology-assisted edu- 
cation (38/184, 21%, 95% CI 15-27%). The participants 
of the studies were predominantly medical students 
(111/184, 60%, 95% CI 53-67%) and resident physicians 
(39/184, 21%, 95% CI 15-28%). Of the resident based 
studies, there was no predominant specialty, with only 
three studies (0.02%, 95% CI 0.003-0.05%) specific to 
Emergency Medicine. 



Discussion 

Technology-assisted education is used in graduated 
medical education across a variety of content areas and 
subject types. Content areas ranged from basic science 
subjects such anatomy and pathology, to clinical medi- 
cine (training in procedures, diagnosis and manage- 
ment), and even to cognitive skills and attitudes [8-13]. 

Computer and Internet-based methods were the most 
commonly used modalities followed by simulation and 
virtual reality. Clinical studies were the most common 
type of study. The most common study subjects were 
medical students followed by residents and attending 
physicians. Only three articles were related to the speci- 
alty of emergency medicine. The majority of articles 
were from authors based in the US and attempted to 
measure gains in knowledge or skills. Many studies 
sought to measure satisfaction and attitudes toward the 
main intervention. The majority of studies that directly 
compared traditional teaching methods to technology- 
assisted education found technology-assisted education 
equal or superior to traditional teaching methods. We 
did not find any particular method or use of technol- 
ogy-assisted education that could be described as a 
"Best Method." Assessment methods commonly used 
included subject self-assessment by survey, computer 
log, a pretest and posttest, and direct observation. 

Technology-assisted education has the potential to 
enrich learning in ways not possible using traditional 
methods of instruction [14]. Technology- assisted educa- 
tion allows individualized self-paced learning, improved 
assessment, evaluation and feedback while increasing 
learner's exposure to other instructional material 
[2,15,16]. Additionally, technology-assisted education 
provides for inherent efficiency in the administration of 
educational material that encompasses development, dis- 
tribution, retrial, storage and communication. The desire 
to harness these advantages and the other useful fea- 
tures of technology-assisted education is a driving force 
behind the efforts of medical educators to determine the 
most effective use technology-assisted education. 

Our study confirms the findings of previous studies 
that technology-assisted education can result in knowl- 
edge improvement [5,6,17]. Eighty-seven (87) articles in 
our study assessed gains by surveying subjects and ask- 
ing for their self-assessment of improvement in knowl- 
edge or skills after exposure to the study method. This 
may be an inaccurate technique to determine the effec- 
tiveness of the teaching method used in the study. Kirk- 
patrick describes a four-level approach to evaluate 
training programs. These levels are: Reaction, Learning, 
Behavior and Results (See Table 4) [18]. A subject's self- 
reported sense of improvement is likely a measure of 
the Reaction level and not a true measure of learning. 



Category 

Basic science 45/184 24, 18-31 

Clinical medicine 125/184 68,61-75 

CME 6/184 3,1-7 

Other 8/1 84 4, 2-8 

Main outcomes studies assessed 

(Some studies assessed multiple 
outcomes) 

Access 38/184 21, 15-27 

Attitude 47/184 26,19-32 

Clinical skills 57/184 31,24-38 

Knowledge 90/184 49,41-56 

Satisfaction 82/184 45,37-52 
Subjects 

Attendings 21/184 11,7-17 

Medical students 111/184 60,53-67 

Nursing 6/184 3, 1-7 

Residents 39/184 21, 15-28 

Combination of subjects 7/184 4,2-8 

Descriptive study data 

Resident specialty 

Emergency medicine 3/39 8, 2-21 

Internal medicine 5/39 13,4-27 

General surgery 10/39 26,13-42 

Pediatrics 6/39 15,6-31 

Radiology 5/39 13,4-27 

Other 3/39 8, 2-21 

Combination of residents 7/39 18,8-34 

Technology used 

(Some studies used multiple technologies) 59/184 32,25-39 

26/184 14,9-20 

CD-ROM/computer based 16/184 9,5-14 

Simulation 94/184 51,44-59 
Virtual reality 
Web based 

Country of origin 

Canada 12/184 6,3-11 

Germany 18/184 10,6-15 

Great Britain 14/184 8,4-12 

United States 105/184 57,50-64 

Others 35/184 16,11-22 
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Table 2 Description of groups 

Group Number/Group Primary method of assessment Comparison with traditional methods 

One 87 Survey/self-reported results None 

Two 18 Before/after measurement None 

Three 79 Objective assessment Yes 

(pretest/posttest, checklist, etc.) 



Student attitude and acceptance of a training method 
are important precursors to the success of any educa- 
tional method. However, studies that relied solely on 
self-assessment to determine the degree of learning may 
have missed the mark and may be of limited value as a 
result. 

The 64 studies that compared traditional teaching 
methods with technology-assisted education used objec- 
tive measurements to determine learning outcomes such 
as a pretest and posttest, checklist and computer log. 
Two-thirds found technology-assisted education super- 
ior to traditional teaching methods. Why or when tech- 
nology-assisted education might be better than 
traditional teaching methods was not always predictable. 
Visualization has been shown to improve learning [19]. 
The teaching of subject matter that consists of complex 
associations or difficult to demonstrate spatial relation- 
ships using standard methods can be enhanced with 
computer-assisted instruction. Some studies we reviewed 
provide insight on this illusive issue. Thatcher compared 
the use of computer-assisted instruction to traditional 
methods to teach medical students about DNA replica- 
tion and found that the computer-assisted instruction 
group performed 22% better on the posttest than the 
traditional textbook group [20]. Thatcher suggested the 
multimedia teaching that was possible with computer- 
assisted instruction enriched learning. Computer- 
assisted instruction allowed the complicated sequence of 
steps and the spatial relationships associated with DNA 
replication to be presented in a three-dimensional for- 
mat, something that was not possible with a two-dimen- 
sional textbook. 

An article we reviewed by Glittenberg and colleagues 
describes the development and use of a three-dimen- 
sional interactive computer-assisted instruction program 
designed to teach students the basics of the human ocu- 
lomotor system [21]. This teaching program included 
information about the main and auxiliary functions of 
each extra-ocular eye muscle, which eye muscles are 
active during any given movement of the eye, the path 
of the oculomotor cranial nerves, the symptoms of cra- 
nial nerve paralysis, as well as symptoms of various neu- 
rological pathologies. The authors compared this 
teaching program to standard teaching methods that 
used textbooks, pictures and diagrams. Formal assess- 
ment methods found that the computer-assisted 



instruction group performed 20% better than the tradi- 
tional teaching group. Glittenberg noted that the com- 
plex material could be demonstrated in a richer fashion 
using computer-assisted instruction than was possible 
with traditional teaching methods and commented 
that... "These findings suggest that high-quality 3D ani- 
mations may help students and physicians, especially 
those with low-spatial abilities, to conceptualize abstract 
topics in medicine and ophthalmology in a way that 
makes it easier for them to understand and remember 
these topics." The conclusions by Thatcher and Glitten- 
berg are supported by Mayer who contends that multi- 
media learning made possible with technology-assisted 
education allows information to be presented to the stu- 
dent using multiple sensory pathways [14]. This aids the 
students' development in understanding the material. 

However, improvement in student performance with 
technology- assisted education was not universal in the 
studies we reviewed. About a third of studies that com- 
pared technology-assisted education with traditional 
teaching methods found no difference in student perfor- 
mance. Again, it was not always clear why these differ- 
ent teaching methods produced the same results. 
Corton and colleagues developed an interactive compu- 
ter-based method to teach pelvic anatomy and com- 
pared it to a conventional paper-based teaching method 
[22]. Study subjects were randomized and pretests, 
posttests and follow-up tests were used to assess learn- 
ing. They found no difference in knowledge gains 
between the technology-assisted education and tradi- 
tional teaching method group despite that fact that most 
students preferred the technology-assisted education 
method. The authors commented that the small number 
of participants (39) and the fact that many participants 
had technical difficulty viewing the animations and 
videos may have impacted the results. In a similar study, 
Forester examined the effects of four supplemental pro- 
grams on learning of gross anatomy [23]. The four sup- 
plemental programs were student teaching assistance, 
direct study, weekly instructor review and a web-based 
anatomy program. There was no significant difference 
between the interventions as all groups showed 
improvement in knowledge compared to controls. Given 
the complex, spatial relationships associated with anat- 
omy, one might have expected the web-based group to 
outperform the other groups. However, there was no 



Table 3 64 comparative articles 



Author Category Outcomes Methods Study 

measured compared characteristics 
A B 



Number 

of 
subjects 
in study 

C 



Study Type of Magnitude Type of Preferred Country 

subjects resident C of benefit % assessment method of study 

D E 



Uranus et al. 

Wehrs 

Casebeer 

Short et al. 
Butzlaff et al. 

Forester et al. 

Krippendorf 

Hudson et al. 

Taradi et al. 

Thatcher 

McNulty 

Noimark et al. 

Leong et al. 

Prinz 

Degnan et al. 

Burgess et al. 

Vivekananda- 
Schmidt et al. 

Callas et al. 
Ganai et al. 



Clinical 
medicine 

Clinical 
medicine 

CME 

CME 

Clinical 
medicine 

Basic 
science 

Basic 
science 

Basic 
science 

Basic 
science 

Basic 
science 

Basic 
science 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
Medicine 



Cs,Ap 
K,At,S,Ap 
K,S,Ap,Ac 
K 

K,Ap,Ac 
K 

K,S,Ap 
!< 

K,At,S,Ap, 
Ac 

K,At,S,Ap 
K 
K 

K,S 
K 
K 

K,At,S,Ap, 
Ac 

K,Cs,At,S,Ap 
S,Ap,Ac 
Cs 



VR to Sim 

Trad to CAI 

Trad to 
WBL 

Trad to CAI 

Trad to 
WBL 

Trad to 
WBL 

Trad to CAI/ 
VR 

Trad to CAI 

Trad to 
WBL 

Trad to CAI 

Trad to CAI 

Trad to 
WBL 

Trad to CAI 

Trad to CAI 

Trad to CAI 

Trad to 
WBL 

Trad to CAI 

Trad to 
WBL 

VR to Trad 



Cs 

C,L 

C,R 

C,R 
C,R 

C 
C,Cs 

C,R 

Cs 

Cs 

Cs 

Cs 

Cs 

C,R 

C,R 

Cs 
C,R,Cs 

Cs 

C,R 



62 

38 

210 

52 
72 

477 

206 

100 

220 

22 

130 

346 

379 

172 



35-1 
903 
19 



Attendings/med 
students 

Attendings 

Attendings 

Attendings 
Attendings 

Med students 

Med students 

Med students 

Med students 

Med students 

Med students 

Med students 

Med students 

Med students 

Med students 

Med students 

Med students 

Med students 

Med students 



NR 

NR 

NR 

NR 
NR 

NR 

NR 

NR 

NR 

NR 

NR 

NR 

NR 

NR 

NR 

NR 

NR 

NR 

NR 



"Sig Better" 
22 
15 

NRA 
NA 

9 

2 

11 

10 

22 

2 

15 
12 

7/16/19* 
NRA 
NRA 
"Sig better" 
NRA 
-10 



DO 
T,S 
T,S 
S 

T,S 
T,S 
T,S 
T,S 
T,S 
T,S 
TS, CL 

T 
T,S 
T,S 
DO.CL 

S 

DO, S 
S 

DO, T 



Technology-VR 

Technology 

Technology 

Standard 
No difference 

Technology 

Technology 

Technology 

Technology 

Technology 

Technology 

Technology 

Technology 

Technology 

Technology 

Technology 

Technology 

Technology 

Technology 



Austria 

Germany 

USA 

USA 
Germany 

USA 

USA 

UK 
Croatia 
USA 
USA 

UK 
USA 
Austria 

UK 

UK 

UK 
USA 
USA 
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Duque et al. 


Clinical 
medicine 


Cs 


Trad to 

VVDL 


Stolz et al. 


Clinical 
medicine 


K,S,Ap,Ac 


Trad to 

\A/RI 
VVDL 


Duque et al. 


Clinical 
medicine 


Cs,At,S 


Trad to 

WP 
VVDL 


Vash et al. 


Clinical 
medicine 


K 


VK to 1 RAD 


Schilling et al. 


Clinical 
n icuiu rtc 


K,S,Ap 


Trad to 
WBL 


Roesch et al. 


Clinical 
medicine 


K,S 


Trad to 

VA/RI 
VVDL 


Ridgway et al. 


Clinical 


K,S,Ap,Ac, 


Trad to 




medicine 


At 
AL 


\A/R 
VVDL 


Qayumi et al. 


Clinical 
medicine 


K,S,Cs,At, 

An 
Ap, 


Trad to CAI 


Shokar et al. 


Clinical 
medicine 


K,Cs 


Trad to 

\A/RI 
VVDL 


Glittenberg et al. 


Clinical 
medicine 


K,At,S 


Trad to CAI 


Friedl et al. 


Clinical 
medicine 


K,Cs 


Trad to CAI 


Engum et al. 


Clinical 
medicine 


K,Cs,S 


Trad to VR 


Hariri et al. 


Basic 
science 


K,S 


Sim to Trad 


Kumar et al. 


Basic 
science 


K,At,S,Ap, 

A(_ 


Trad to CAI 


Cox et al. 


Clinical 
medicine 


K,At,S 


Trad to 

\,VP 
VVDL 


Chou et al. 


Clinical 
medicine 


K 


VR to Trad 


Hahne et al. 


Clinical 
medicine 


K,At,S 


Trad to CAI 


Curran et al. 


Clinical 
medicine 


K,Cs,At,S 


Sim to Trad 


Wahlgren et al. 


Clinical 
medicine 


K,At,S,Ap, 
Ac 


Trad to CAI 


Raij et al. 


Clinical 
medicine 


Cs,Ap 


VR to Trad 


Nackman et al. 


Clinical 
medicine 


Cs 


Trad to 

WBL 



C,L 

C,R 
C,R 
C,R 
L 

Cs 
C,R 
C 

C,R 
C 

C,R 
C,R 

Cs 
C,R 
C,R 
C,R 
C,R 
C,R,L 

C 

Cs 



100 
MR 
133 
48 
238 
3050 



179 
140 
195 
163 
29 
212 
100 
16 
167 
60 
116 
82 



Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 
Med students 



MR 
MR 
MR 
MR 
NR 
MR 
NR 
NR 
NR 
NR 
Surg 
NR 
NR 
NR 
NR 
NR 
NR 
NR 
NR 
NR 
NR 



NRA 

10 

14 
NRA 
26/46* 

10 
4/10* 

Sig 

improvement 
5/4* 

17 

15/18 Cs 
only 

NRA 
NA 
NRA 
NA 
NA 
NA 
NA 
NA 
NA 
NA 



T,S Technology Canada 

T,S Technology Germany 

T.S.CKL Technology Canada 

T Technology (in 1/4 Iran 
content areas) 

T,S Technology USA 

T,S Technology Germany 

T,S, CL Technology Ireland 

T,S,DO Technology Japan 

T,DO Technology USA 

T,S Technology Austria 

T,DO Standard (no diff K, CAI Germany 
> Cs) 

T,S, CKL Standard USA 

T,S No difference USA 

S Technology Australia 

T,S No difference USA 

CKL No difference USA 

T,S No difference Germany 

T,S, CKL No difference Canada 

T,S, CL No difference Sweden 

S,CL No difference USA 

T No difference USA 
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Table 3 64 comparative articles (Continued) 



Karnath et al. 

Feeg et al. 
Anderson et al. 

Dee et al. 
Errichetti et al. 

Ryan et al. 

Gold et al. 

Roche et al. 

Gold et al. 
Barsuck et al. 

Xiao et al. 

Knoll et al. 
Mahnke et al. 

Park et al. 
Schijven et al. 

Maiss et al. 

Jonas et al. 
Sedlack et al. 
Corton et al. 
Jowett et al. 
Chung et al. 



Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Basic 
science 

Clinical 
medicine 

Basic 
science 

Basic 
science 

Basic 
science 

Basic 
ccience 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Clinical 
medicine 

Basic 
science 

Clinical 
medicine 

Clinical 
medicine 



Cs 
K,At 
K,Cs 

K 

Cs 

K 

K,S 
K 

K,S 
K,Cs 
K,Cs 

Cs 

Cs 

Cs 
K,S,Ap 

Cs 

Cs 

Cs 



Trad to CAI 
Trad to CAI 

Trad to 

WBL 

Trad to CAI 

Trad to CAI 

Trad to CAI 

Trad to CAI 

Trad to CAI 

Trad to CAI 

Sim to Trad 

Trad to CAI/ 
WBL 

Sim to Trad 

Trad to CAI 

Sim to Trad 

VR to Trad 

Sim to 
Other 

Sim to Trad 
Sim to Trad 



K,S,Ap Trad to CAI 



Cs 
K 



Trad to CAI 

Trad to 
WBL 



C,R 
C,R 
Cs 
L 

C,R 
C 

C,R 
C,R 
C,R 
Cs 
C,R,Cs 
C,R 
Cs 
C,R 
C 

C,R 
C,R 
C 
C,R,L 
C,R,L 
C,R,L 



192 
125 
180 
98 
40 
23 
138 
38 
138 
72 
50 
30 
40 
24 
24 
35 
1-1 
38 
39 
30 
63 



Med students 

Nurses 

Nurses 

Other 

Med students 

Med students 

Residents 

Residents 

Residents 

Residents 

Residents 

Residents/ 
attendings 

Residents 

Residents 

Residents 

Residents 

Residents/med 
students 

Residents 
Residents 
Residents 
Residents 



MR 

MR 

MR 

MR 

NR 

MR 
Surg 
Peds 
Surg 

MR 
EM/Surg 

GU 
Peds 
IM/Surg 
Surg 

IM 
Ophthal 

IM 
GYN 
Surg 

EM 



NA 

10 

MA 
NRA 
NRA 

NA 
NRA 

11 

NRA 
47/31* 
-11 

NRA 
21 

NRA 
NRA 
NRA 
NRA 
NRA 

NA 

NA 

NA 



T 
T 

S 
S 

CL 
S 

TS.CL 

T,S 
T,S,CL 
TCL 

CL 
CLCKL 

T 

CLCKL 
DO, S 

CL 

CKL 
S.CKL 

T,S 

T 

TS.CL 



No difference 
Technology 
No difference 
Technology 
Technology 
No difference 
Technology 
Technology 
Technology 
Technology 
Technology 
Technology 
Technology 
Technology 
Technology 
Technology 
Technology 
Technology 
No difference 
No difference 
No difference 



USA 
USA 
USA 
USA 
USA 
Ireland 
USA 
USA 
USA 
Israel 
USA 
Germany 

USA 
Canada 
Netherlands 
France 
Germany 
USA 
USA 
Canada 
USA 
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Table 3 64 comparative articles (Continued) 



Davis et al. 


Clinical 
medicine 


K,At 


Trad to CAI 


C,R 


55 


Residents 


NR residents 


NA 


T,S 


No difference 


UK 


Ferguson et al. 


Clinical 
medicine 


K 


Trad to 
WBL 


0 


19 


Residents 


Surg 


NA 


TCL 


No difference 


USA 


Bridgemohan et al. 


Clinical 
medicine 


K,At,S 


Trad to CAI 


R 


46 


Residents 


Peds 


NA 


T,S 


No difference 


USA 



Legend: outcomes measured A 

Ac: Access (can students get to and use the learning material); Ap:Applicability (can you teach with this method); At: Attitude; Cs: Clinical Skills (e.g.: H+P skills, EKG, CXR interpretation); Cs: Cost; K: Knowledge retention/ 
learning; O: Other; S: Satisfaction. 
Legend: study characteristics B 

C: Controlled (there was a control group); Cs: Cross sectional (single point in time); L: Longitudinal (more than one point in time); 0: Other; R: Random (randomization of groups.) 
Legend: number of subjects in study/type of resident C 
NR: Not reported 

Legend: magnitude of benefit % D 

NRA: Not readily available; NA: Not applicable; *Multiple measurements 
Legend: type of assessment E 

DO: Direct observation; T: Test; S: Survey; CL: Computer log; CKL: Check list. 
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Table 4 Kirkpatrick's four levels of evaluation 

Level Focus of level 

Reaction Student's perception of or satisfaction with training 
method 

Learning To measure if students' knowledge/skills/attitude 
changed 

Behavior Determine if the new knowledge/skills/attitudes are 
being used by the student 

Results The trainings impact on the organization 



Adapted from Kirkpatrick [18] 

explanation advanced as to why all four groups per- 
formed the same. Interestingly, students in this study 
preferred the method that provided more direct contact 
with the instructors. 

Other investigators in our review, such as Davis et al., 
who conducted a study on teaching evidence-based 
medicine, and Cox et al., who studied teaching concepts 
related to the underserved, found no difference between 
technology-assisted education and traditional teaching 
methods, suggest that technology-assisted education 
methods could serve as a possible alternative to lecture 
[24,25]. These authors note the potential savings in time 
related to student/instructor travel and preparation of 
content as well as the ability to standardize content and 
teaching methods [24,25]. An additional advantage of 
the technology-assisted education methods is that these 
methods can be made available continuously for use 
when convenient to the students. Whether "no differ- 
ence" means that instructional methods are interchange- 
able is an open question that is probably best 
determined by further study. 

We were unable to identify specific information in the 
articles we reviewed that lead us to a "Best Method or 
Best Use" for technology-assisted education. We had 
hoped that the 64 studies that directly compared technol- 
ogy-assisted education to traditional education methods 
would provide information regarding this question. In 
many of these reviewed studies, authors offered opinions 
similar to those advanced by Thatcher and Glittenberg 
within their papers. Additional light is shed on the issue 
by other investigators. Cook et al. published an article that 
reviews ten steps to effective web-based learning [26]. 
Issenberg and colleagues, in a systemic review of simula- 
tion-based education, identified 20 important guidelines 
they recommend authors should adhere to when conduct- 
ing research on simulation-based education [27]. The 
efforts of these authors provide more pieces to the puzzle 
that help bring the answer to the question of how best to 
use technology-assisted education into better focus. 
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When motion pictures were first invented, Thomas 
Edison is reported to have predicted that motion pic- 
tures would revolutionize education. Experts agree it did 
not [14]. Similar unfulfilled claims were made when 
other technologies like radio and television were 
invented [14]. We certainly should not repeat the mis- 
takes of our predecessors. Our study found that technol- 
ogy-assisted education is used across the wide spectrum 
of graduate medical education underscoring the reality 
that technology-assisted education is here to stay and 
will likely change teaching and learning in ways we can- 
not predict. Consider the experience of Piemme, who in 
a 1988 article expressed excitement at the potential uses 
of a new technology called the CD-ROM [1]. 

Despite a body of research that suggests technology- 
assisted education can improve knowledge gains and 
student achievement, there remains difficulty in estab- 
lishing technology-assisted education's exact role in the 
current curriculum and the degree to which it can 
replace traditional teaching methods [5-7,15,28]. Rapidly 
evolving computer technology presents educators with 
potential new methods of instruction on a near continu- 
ous basis [1,29,30]. This is one factor that makes it diffi- 
cult to determine the best use of technology-assisted 
education. An additional confounder may be faulty use 
of technology-assisted education by educators. We have 
been constantly reminded that technology-assisted edu- 
cation is a tool that needs to be used properly if it is to 
be effective. Educators should seek resources that 
explain how to effectively use technology-assisted educa- 
tion before investing time and money on its application 
(see Additional readings) [26,27,31-34]. 

Another barrier to determining technology-assisted 
education's role in the curriculum is the quality of the 
published research in this domain. Our study is similar 
to other studies that showed conflicting results when 
technology-assisted education methods were compared 
to traditional methods [6,17]. In the studies we 
reviewed, there was a wide variety of subjects, settings 



Possible measurement method 

Survey, focus groups 

Control group 

Objective pretest/post test of knowledge/skills 
Direct observation Checklist 

Control group 

Direct observation checklist 

Before/after interview or survey of student's direct contacts or supervisors 

An improvement in quality, productivity, reduction in cost, increase profit or some 
other tangible benefit to the organization 
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and assessment methods. Many investigators have 
voiced concerns about the persistent limitations, often 
systemic, that exist in studies on computer-based educa- 
tion [5-7,17,35]. Some authors have suggested that stu- 
dies that compare traditional non-computer teaching 
methods to computer-based learning are fundamentally 
flawed because no true comparison exists between the 
two [36,37]. Friedman and Cook suggest it would be 
more productive to study one form of technology- 
assisted education versus another or study how to effec- 
tively integrate and measure technology-assisted educa- 
tion's impact on learning [36,37]. 

We had difficulty identifying the core elements in 
many of the studies we reviewed (see Limitations). Con- 
cordant with this finding, a number of authors have 
recommended a more organized and programmatic 
approach to research in the area of technology-assisted 
education (and medical education in general) to advance 
our understanding in this domain [36-39]. Detailed 
information about all aspects of the study as well as spe- 
cific goals and objectives combined with formal, valid 
assessment methods are critical precursors for an effec- 
tive study. 

Medical educators are tasked with teaching competen- 
cies established by the Accreditation Council of Graduate 
Medical Education (ACGME). These competencies have 
been described elsewhere [40]. We would recommend 
authors align their research with the ACGME core com- 
petencies and identify which core competencies are being 
addressed by the study. Competency can be defined as 
the specific knowledge, skill and attitude needed to com- 
plete a task correctly. Authors should identify which 
aspect of competency their study deals with (knowledge, 
skill or attitude) and precisely how the competency is 
measured. A more organized approach to technology- 
assisted education research may allow educators across 
specialties to learn from each other. This could facilitate 
a more comprehensive cross-specialty understanding of 
how to best use technology-assisted education. If, for 
example, researchers in the surgical specialties identified 
key training elements of technology-assisted education 
need to master the skills of laparoscopic procedures, 
these training elements might be important in learning 
other skills such as endotracheal intubation or lumbar 
puncture. Similarly, if one specialty identified the key 
training elements of professionalism that could be taught 
or measured using technology- assisted education, all spe- 
cialties would benefit from this knowledge. More struc- 
tured, programmatic research maybe the best way to 
foster transfer of training knowledge from one specialty 
to another and may be the only way to identify a "Best 
Method or Best Use" for technology-assisted education, 
something that has eluded medical educators for decades. 



Limitations 

Our study has a number of limitations. We attempted to 
conduct a review of the medical literature covering a 5- 
year period. The sensitivity of literature searches varies 
and can be improved when conducted by librarians [41]. 
Despite using the services of experienced research 
librarians for our search, some articles may not have 
been identified and therefore are not included in the 
body of literature. 

The wide variety of study designs, settings, subjects, 
assessment and reporting methods made combining 
results impossible. This heterogeneity of outcomes pre- 
cluded a meta-analysis from being performed [42]. 
Additionally, reporting methods used in some studies 
made data abstraction difficult. In some studies, it was 
difficult to determine core elements such as who the 
subjects were because of vague descriptions and incom- 
plete definitions. Despite the use of a study panel to 
resolve disagreements associated with data abstraction, 
errors could have been made. Additionally, we did not 
independently evaluate the original author's results as 
this was beyond the scope of our review. 

Conclusions 

Technology-assisted education is used in graduate medi- 
cal education across a variety of content areas and sub- 
ject types. Studies in our review show technology- 
assisted education can result in improvements in knowl- 
edge. Sixty-seven percent (67%) of studies that directly 
compared knowledge gains in technology-assisted edu- 
cation to traditional teaching methods found technol- 
ogy-assisted education equal or superior to traditional 
teaching methods. Only three articles dealt primarily 
with EM. This suggests further research in our specialty 
is warranted. We would recommended EM educators 
follow programmatic research methods to avoid limita- 
tions found in other studies and consider aligning their 
research with the ACGME core competencies. 

Article Summary Key Questions 

Why is the topic important? 

Technology-assisted education is widely used in 
graduate medical education. Technology-assisted 
education needs to be used correctly if it is to be 
effective. Otherwise, valuable training time and 
resources could be wasted. 



What does this study attempt to show? 

Our study attempted to determine the scope of use 
of technology-assisted education, whether technol- 
ogy-assisted education improved knowledge when 
compared with traditional teaching methods, and 
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whether a "Best Method or Best Use" for technol- 
ogy-assisted education could be identified. 

What are the key findings? 

-Technology-assisted education can improve knowl- 
edge. However, use of technology- assisted education 
does not guarantee knowledge gains as approxi- 
mately one third of studies did not show improve- 
ment in knowledge gains. 

-Many articles in our study (87) assessed gains by 
surveying subjects and asking for their self-assess- 
ment of improvement in knowledge or skills after 
exposure to the study method. This may be an inac- 
curate technique to determine the effectiveness of 
the teaching method used. 

- Despite years of use and multiple studies, a "Best 
Method" or "Best Use" of technology-assisted educa- 
tion was not found. 

- Only three articles dealt primarily with Emergency 
Medicine. This suggests further research in our spe- 
cialty is warranted. We would recommended EM 
educators follow programmatic research methods to 
avoid limitations found in other studies and consider 
aligning their research with the ACGME core 
competencies. 

How is patient care impacted? 

Improvements in instructional methods should result 
in enhanced competency by physician thereby 
improving patient care. 
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